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The  NATO  Science  and  Technology  Organization 


Science  &  Technology  (S&T)  in  the  NATO  context  is  defined  as  the  selective  and  rigorous  generation  and  application  of 
state-of-the-art,  validated  knowledge  for  defence  and  security  purposes.  S&T  activities  embrace  scientific  research, 
technology  development,  transition,  application  and  field-testing,  experimentation  and  a  range  of  related  scientific 
activities  that  include  systems  engineering,  operational  research  and  analysis,  synthesis,  integration  and  validation  of 
knowledge  derived  through  the  scientific  method. 

In  NATO,  S&T  is  addressed  using  different  business  models,  namely  a  collaborative  business  model  where  NATO 
provides  a  forum  where  NATO  Nations  and  partner  Nations  elect  to  use  their  national  resources  to  define,  conduct  and 
promote  cooperative  research  and  information  exchange,  and  secondly  an  in-house  delivery  business  model  where  S&T 
activities  are  conducted  in  a  NATO  dedicated  executive  body,  having  its  own  personnel,  capabilities  and  infrastructure. 

The  mission  of  the  NATO  Science  &  Technology  Organization  (STO)  is  to  help  position  the  Nations’  and  NATO’s  S&T 
investments  as  a  strategic  enabler  of  the  knowledge  and  technology  advantage  for  the  defence  and  security  posture  of 
NATO  Nations  and  partner  Nations,  by  conducting  and  promoting  S&T  activities  that  augment  and  leverage  the 
capabilities  and  programmes  of  the  Alliance,  of  the  NATO  Nations  and  the  partner  Nations,  in  support  of  NATO’s 
objectives,  and  contributing  to  NATO’s  ability  to  enable  and  influence  security  and  defence  related  capability 
development  and  threat  mitigation  in  NATO  Nations  and  partner  Nations,  in  accordance  with  NATO  policies. 


The  total  spectrum  of  this  collaborative  effort  is  addressed  by  six  Technical  Panels  who  manage  a  wide  range  of 
scientific  research  activities,  a  Group  specialising  in  modelling  and  simulation,  plus  a  Committee  dedicated  to 
supporting  the  information  management  needs  of  the  organization. 

AVT  Applied  Vehicle  Technology  Panel 

HFM  Human  Factors  and  Medicine  Panel 

1ST  Information  Systems  Technology  Panel 

NMSG  NATO  Modelling  and  Simulation  Group 

SAS  System  Analysis  and  Studies  Panel 

SCI  Systems  Concepts  and  Integration  Panel 

SET  Sensors  and  Electronics  Technology  Panel 


These  Panels  and  Group  are  the  power-house  of  the  collaborative  model  and  are  made  up  of  national  representatives  as 
well  as  recognised  world-class  scientists,  engineers  and  information  specialists.  In  addition  to  providing  critical 
technical  oversight,  they  also  provide  a  communication  link  to  military  users  and  other  NATO  bodies. 


The  scientific  and  technological  work  is  carried  out  by  Technical  Teams,  created  under  one  or  more  of  these  eight 
bodies,  for  specific  research  activities  which  have  a  defined  duration.  These  research  activities  can  take  a  variety  of 
forms,  including  Task  Groups,  Workshops,  Symposia,  Specialists’  Meetings,  Lecture  Series  and  Technical  Courses. 


The  content  of  this  publication  has  been  reproduced  directly  from  material  supplied  by  STO  or  the  authors. 
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Advancements  in  Distributed  Learning  (ADL) 
Environment  in  Support  of  Transformation 

(STO-TR-HFM-212) 

Executive  Summary 

Multi-national  partners  identified  the  need  for  a  number  of  technology  advancements  and  the  need  to  address 
current  and  future  education  and  training  shortfalls.  It  was  recommended  to  review  the  ADL  needs  of 
NATO,  PfP,  MD,  and  other  Coalition  Partners  and  determine  what  advancements  were  available  and  which 
ones  were  undergoing  research  and  development,  to  leverage  global  best  practices  of  ADL  to  improve 
readiness  of  military  and  civilian  personnel. 

The  main  objective  of  RTG  HFM-212  was  to  identify  and  explore  operational  needs  and  technology 
capabilities  for  an  agile,  open  ADL  framework  for  content  and  infrastructure  that  can  be  leveraged  for  global 
collaboration  across  NATO,  PfP,  MD,  and  other  Coalition  Partners.  Additionally,  to  explore  the  continued 
effectiveness  of  the  ADL  approach  and  its  capabilities  to  assess  and  track  learner  success  while  utilizing 
advanced  distributed  learning  technologies  (i.e.,  conventional,  traditional  and  emerging). 

RTG  HFM-212  enjoyed  continual  success  in  reaching  its  stated  objectives.  It  utilized  an  existing  NATO 
infrastructure,  actively  collaborated  with  other  NATO  assets  (e.g.,  the  NATO  Training  Group  and  the  PfP 
Consortium),  and  built  a  transition  path  toward  the  delivery  of  its  work  to  NATO  school  houses. 
By  accomplishing  the  RTG  goals  of  creating  and  strengthening  partnerships  among  24  key  Partner  Nations, 
RTG  HFM-212  has  contributed  sizable  technology  awareness  as  well  as  real  capabilities  to  the  NATO 
education  and  training  communities. 

A  number  of  important  projects  were  motivated  by  the  efforts  of  this  RTG.  These  projects  are  described  in 
Section  6.0.  These  projects  have  the  potential  for  large-scale  use  by  NATO  and  Partner  Nations.  The  way 
ahead  for  research  in  this  area  is  reflected  in  the  proposed,  but  unfunded,  Multi-channel  Learning  System 
(MLS)  project.  Where  this  project  focused  on  identifying  the  best  methods  for  providing  distance  education 
for  international  military  students  preparing  for  resident  training  in  the  U.S.,  its  transition  to  the  NTG  IT&ED 
allows  it  to  continue  contributing  to  NATO  education  and  training  schoolhouses.  Through  this  broad 
collaboration  within  the  Alliance,  RTG  HFM-212  was  able  to: 

1)  Develop  a  capability  that  ensures  International  Military  Student  Pre-Departure  Brief  (1MSPDB) 
students  have  a  positive  and  successful  experience  in  the  U.S.  along  with  their  U.S.  counterparts; 

2)  Identify  the  best  methods  for  providing  distance  education  for  international  military  students;  and 

3)  Evaluate  the  effectiveness  of  multiple  learning  formats  to  support  Security  Cooperation  Education 
and  Training  Program  (SCETP)  requirements. 

Finally,  RTG  HFM-212  initiated  application  of  ADL  capabilities  in  the  medical  and  health  sciences 
community  via  its  sponsor,  the  NATO  Human  Factors  and  Medicine  Panel. 
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Progres  en  apprentissage  distribue  (ADL) 
a  l’appui  de  la  transformation 

(STO-TR-HFM-212) 

Synthese 

Les  partenaires  des  differents  pays  ont  remarque  qu’un  certain  nombre  de  progres  technologiques  etaient 
necessaires  et  qu’il  fallait  corriger  des  lacunes  actuelles  et  a  venir  en  matiere  d’education  et  de  formation.  II  a 
ete  recommande  d’examiner  les  besoins  d’ADL  de  l’OTAN,  du  PpP,  du  DM  et  des  autres  partenaires  de 
coalition  et  de  determiner  les  avancees  disponibles  et  celles  faisant  l’objet  de  recherche  et  developpement, 
afin  de  tirer  parti  des  meilleures  pratiques  mondiales  d’ADL  pour  ameliorer  l’etat  de  preparation  du 
personnel  militaire  et  civil. 

Le  principal  objectif  du  RTG  HFM-212  etait  d’identifier  et  d’etudier  les  besoins  operationnels  et  les 
capacites  technologiques  en  vue  d’un  cadre  d’ADL  agile  et  ouvert  qui  offre  un  contenu  et  une  infrastructure 
permettant  une  collaboration  mondiale  dans  Pensemble  de  l’OTAN,  du  PpP,  du  DM  et  des  autres  partenaires 
de  coalition.  II  s’agissait  de  plus  d’etudier  l’efficacite  continue  de  la  demarche  d’ADL  et  ses  capacites  a 
evaluer  et  suivre  la  reussite  des  eleves,  tout  en  utilisant  des  technologies  d’apprentissage  avancees  et 
reparties  (autrement  dit,  classiques,  traditionnelles  et  emergentes). 

Le  RTG  HFM-212  a  constamment  reussi  a  atteindre  les  objectifs  qui  lui  avaient  ete  assignes.  Le  RTG 
HFM-212  a  utilise  une  infrastructure  existante  de  l’OTAN,  a  collabore  activement  avec  d’autres  elements  de 
l’OTAN  (tels  que  le  Groupe  OTAN  pour  l’entrainement  et  le  consortium  du  PpP)  et  a  ouvert  la  voie  a  la 
transmission  de  ses  travaux  aux  ecoles  de  l’OTAN.  En  atteignant  les  objectifs  qui  lui  avaient  ete  attribues, 
a  savoir  la  creation  et  le  renforcement  des  partenariats  entre  24  pays  partenaires  essentiels,  le  RTG  HFM-212 
a  foumi  de  reelles  capacites  et  une  connaissance  mesurable  de  la  technologie  aux  communautes  de 
l’enseignement  et  de  la  formation. 

Un  certain  nombre  de  projets  importants  ont  ete  motives  par  les  travaux  de  ce  RTG.  Ils  sont  decrits  a  la 
section  6.0.  L’OTAN  et  ses  pays  partenaires  peuvent  utiliser  ces  projets  a  grande  echelle.  La  marche  a  suivre 
pour  la  recherche  dans  ce  domaine  se  reflete  dans  le  projet  de  systeme  d’apprentissage  multicanal 
(MLS,  Multi-channel  Learning  System)  propose,  mais  non  finance.  Alors  que  le  present  projet  s’est  concentre 
sur  1’ identification  des  meilleures  methodes  d’enseignement  a  distance  pour  les  eleves  militaires 
intemationaux  se  preparant  a  un  intemat  aux  Etats-Unis,  sa  transition  vers  le  NTG  IT&ED  lui  permet  de 
continuer  a  contribuer  a  l’enseignement  et  aux  ecoles  de  formation  de  l’OTAN.  Par  le  biais  de  cette  vaste 
collaboration  au  sein  de  l’Alliance,  le  RTG  HFM-212  a  : 

1)  Developpe  une  capacite  qui  garantit  que  les  eleves  suivant  le  stage  pre-depart  International  Military 
Student  Pre-Departure  Brief  (IMSPDB)  beneficient  d’une  experience  positive  et  reussie  aux 
Etats-Unis,  aux  cotes  de  leurs  homologues  americains  ; 

2)  Identifie  les  meilleures  methodes  d’enseignement  a  distance  pour  les  eleves  militaires  intemationaux  ; 
et 

3)  Evalue  l’efficacite  des  multiples  formats  d’apprentissage  a  l’appui  des  exigences  du  Security 
Cooperation  Education  and  Training  Program  (SCETP). 

Enfin,  le  RTG  HFM-212  a  commence  a  appliquer  les  capacites  ADL  dans  la  communaute  medicale  et  des 
sciences  de  la  sante  par  le  biais  de  son  commanditaire,  la  Commission  sur  les  facteurs  humains  et  la 
medecine  de  l’OTAN. 


ES  -2 


STO-TR-HFM-212 


organization 


ADVANCEMENTS  IN  DISTRIBUTED  LEARNING  (ADL) 
ENVIRONMENT  IN  SUPPORT  OF  TRANSFORMATION 

1.0  ORIGIN  OF  RTG  HFM-212 
1.1  Background 

International  advancements  in  Distributed  Learning  (DL)  have  accelerated  over  the  past  decades  to  meet 
continuing  user  needs.  These  needs  often  responded  to  technological  advances  in  networking,  computer  science, 
learning  science,  and  the  knowledge,  experience,  and  expectations  of  burgeoning  user  populations.  Building 
on  technological  advances,  the  USA  and  its  multi-national  learning  technology  colleagues  encouraged  the 
NATO  Alliance  in  the  1990s  to  emphasize  distributed  learning  as  a  means  for  PfP  Nations  to  reach  out  via 
communications  and  collaborations. 

NATO’s  work  in  distributed  learning  quickly  took  advantage  of  the  rapidly  developing  Advanced  Distributed 
Learning  (ADL)  initiative  in  the  USA.  One  of  the  major  issues  they  faced  was  how  to  collaborate  and 
communicate  over  a  computer  network  that  might  not  exist  in  many  Nations.  The  Alliance  decided  to  leverage  a 
newly  developed  PfP  computer  network  as  a  backbone  for  early  NATO  ADL  development,  collaboration  and 
evaluation  of  sharable  learning  content.  This  decision,  which  was  discussed  at  NATO/PfP  meeting  circa  1 997  in 
Paris,  France,  initiated  a  new  management  infrastructure  for  collaboration  and  integration  of  learning  courses 
and  technologies.  This  ADL  effort  was  soon  expanded  in  both  content  and  breadth  to  what  became  known  as 
the  PfP  Consortium.  As  this  enterprise  organizational  structure  grew  and  matured,  it  spawned  a  number  of 
international  activities  to  explore  and  capitalize  on  the  art  of  the  possible,  change  the  way  we  learn  in  general, 
and  enhance  NATO  and  its  Alliance  partners’  mission  effectiveness  in  particular. 

It  is  important  to  operationally  define  the  major  components  of  ADL.  “Learning”  in  the  ADL  sense  was  used  as 
a  catch-all  designator  for  education,  training,  and  performance  /  decision  aiding.  “Distributed”  in  ADL  signifies 
learning  that  can  be  provided  in  classrooms  with  a  teacher  present,  in  the  field  linking  together  widely  dispersed 
instructors  and  students,  or  standing  alone  with  no  instructor  other  than  the  computer  itself.  “Advanced”  in  ADL 
implies  affordable,  interactive,  adaptive,  on-demand  instruction  delivered  using  computer  technology  so  that  it  is 
available  anytime,  anywhere. 

Empirical  research,  available  as  early  as  the  1960s,  suggested  the  feasibility  of  ADL  capability  with  its  goals  of 
developing  and  implementing: 

•  Individualized,  tutorial  Teaming’  (including  individualized  performance  /  decision  aiding)  that  can  be 
provided  affordably  by  technology-based  learning. 

•  Technology-based  learning  that  can  be  more  effective  and  can  produce  greater  return  on  investment  than 
conventional  instructional  approaches  across  many  instructional  objectives  and  subject  matters. 

•  Technology-based  learning  that  allows  education,  training,  performance  aiding,  and  decision  aiding  to 
be  delivered  on  platforms  ranging  from  hand-held  devices,  to  desk-top  computers,  to  capabilities 
embedded  in  operational  equipment. 

Statistical  findings  from  this  research  may  be  summarized  by  a  “Rule  of  Thirds”  discussed  in  Annex  A. 
It  suggests  that  one  can  either  reduce  instructional  time  to  reach  instructional  goals  by  about  one-third, 
or  increase  the  skills  and  knowledge  acquired  by  about  one -third  while  holding  instructional  time  constant. 
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Development  of  the  Partnership  for  Peace  Consortium  (PfPC)  was  fundamental  to  NATO  ADL.  The  PfPC  is  an 
international  security  cooperation  organization  of  over  800  defence  academies  and  security  studies  institutes 
across  60  countries.  Officially  founded  in  1999  during  the  NATO  Summit  in  Washington,  DC,  USA,  the  PfPC 
was  chartered  to  promote  defence  institution  building  and  foster  regional  stability  and  security  cooperation 
through  multi-national  defence  education  and  research,  which  the  PfPC  accomplishes  via  its  international 
network  of  educators  and  researchers. 

Thus  the  creation  of  the  NATO  ADL  RTG  HFM-212  was  founded  on  a  vision  of  how  and  where  learning  could 
advance  human  performance,  coordination,  and  mission  effectiveness  for  the  Alliance  during  the  years  to  come. 
It  was  based  on  the  strong  foundation  established  by  the  PfPC  and  its  infrastructure  of  Member  Nations  to  focus 
on  education  and  training  as  a  primary  NATO  mission.  The  long-term  vision  for  ADL  is  an  extrapolation  from 
such  technological  developments  as  portable,  increasingly  accessible  computing,  the  global  information 
infrastructure,  modular  object-oriented  architectures,  and  natural  language  processing.  The  march  toward  devices 
that  might  be  described  as  personal  learning  associates  seems  inevitable.  These  portable  devices  will  act  as 
personal  accessories  for  users.  They  will  respond  on  demand  and  in  real  time  to  user  needs  for  education, 
training,  and  performance  aiding  by  assembling  relevant  objects  from  the  global  infrastructure  and  engaging  the 
user  in  guided  conversations.  This  approach  can  substantially  enhance  the  knowledge,  skills,  and/or  problem¬ 
solving  capabilities  of  individuals  and/or  groups  of  dispersed  individuals  whose  devices  are  wirelessly  linked 
together. 

Based  on  a  PfP  initiative,  in  March  2008,  a  Multi-national  Virtual  Learning  Environment  (MVLE)  Proof  of 
Concept  was  conducted  by  the  Commander  Naval  Forces  Europe/Commander  North  Africa/Commander  SDCTH 
Fleet.  It  was  based  on  an  emergent  requirement  to  establish  an  advanced  distance/distributed  learning  program 
that  would  assist  key  regions  of  Europe  and  Africa  in  training  future  military  and  civilian  leaders.  The  goal  was 
to  create  and  strengthen  partnerships  in  key  Nations.  It  succeeded  far  beyond  its  expectations. 

The  initial  MVLE  Proof  of  Concept  Operational  Evaluation  was  held  at  the  Office  of  Naval  Research  Global 
(ONRG)  in  London,  England,  with  35  participants  from  government  and  commercial  organizations  from 
Azerbaijan,  Georgia,  Germany,  Moldova,  Romania,  the  United  Kingdom  and  the  United  States.  By  the  time  the 
Operational  Evaluation  had  been  completed,  seventy-two  (72)  participants  from  Azerbaijan,  Bulgaria,  Georgia, 
Germany,  Moldova,  Romania,  Tadzhikistan,  the  United  Kingdom  and  the  United  States  had  participated  in  an 
assessment  of  a  technology-sound  Advanced  Distributed  Learning  capability. 

As  a  result  of  this  MVLE  Proof  of  Concept  Operational  Evaluation  meeting,  multi-national  partners  identified 
the  need  for  a  number  of  technology  advancements  and  the  need  to  address  current  and  future  education  and 
training  shortfalls.  It  was  recommended  that  the  NATO  RTO  Human  Factors  and  Medical  (HFM)  Panel  establish 
a  Research  Technology  Group  (RTG)  to  review  the  ADL  needs  of  NATO,  PfP,  MD,  and  other  Coalition  Partners 
and  determine  what  advancements  were  available  for  immediate  use  and  which  ones  were  undergoing  research 
and  development. 

Working  with  NATO  Training  Group  (NTG)  leaders  in  the  Individual  Training  and  Educational  Development 
(1T&ED)  Working  Group,  NATO  HFM  Panel  established  ET-105  which  transitions  into  RTG  LIFM-212.  This 
NATO  NTG/RTO  team  has  shared  and  transitioned  its  knowledge  throughout  the  Alliance  with  the  help  of  the 
Allied  Command  Transformation  (ACT)  and  Mr.  Paul  Thurkettle.  Working  collaboratively  with  the  NATO 
Science  and  Technology  Office  in  Paris,  RTG  LIFM-212  continues  to  contribute  to  NATO’s  Training 
infrastructure.  Some  actions  and  activities  now  taking  place  are: 

•  Defence  education  curriculum  development; 

•  Education  delivery  methods  (e.g.,  distributed  computer-based  learning); 
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•  Foreign  policy  recommendation  papers; 

•  Information  sharing;  and 

•  Coordination  of  skills  and  assets. 

The  Technical  Activity  Program  (TAP)  for  this  Human  Factors  and  Medical  (HFM)  Research  Technology 
Group  (RTG)  212  was  approved  in  2010  with  an  end  date  of  December  2013.  Final  Report  was  due  Fall  of  2014. 
However  due  to  a  number  of  funding,  organizational,  and  resultant  travel  restrictions,  the  final  report  was 
delayed  until  the  first  quarter  of  2015. 

1.2  Justification  (Relevance  for  NATO) 

The  significant  goal  of  ADL  to  NATO  is  to  enable  and  leverage  global  best  practices  of  ADL  for  providing  more 
effectiveness  during  NATO  missions  and  operations.  Through  such  leveraging,  the  coalition  will  be  able  to 
improve  readiness  of  military  and  civilian  personnel  anywhere  and  anytime  the  need  arises.  ADL  topics 
addressed  and  discussed  by  RTG  HFM-212  were  efficient  and  interoperable  sharing  of  content,  competencies, 
utilization  of  emerging  learning  concepts  and  technologies  to  enhance  training,  and  to  develop  and  provide 
meaningful  performance  metrics. 

This  RTG  found  considerable  evidence  (data)  suggesting  that  computer  technology  might  be  affecting  this  third 
revolution.  Some  key  findings  from  the  last  three  years  of  RTG  HFM-212  meeting  can  be  summarized  as 
follows: 


•  Although  individualized  learning  tailored  to  the  background  and  needs  of  individual  students  has  long 
been  viewed  as  an  imperative,  it  has  also  been  viewed  as  unaffordable  [40].  With  few  exceptions, 
we  cannot  afford  one  instructor  for  every  student  -  an  Aristotle  for  every  Alexander.  Computer 
technology  can  make  this  imperative  affordable.  A  core  argument  for  ADL,  then,  is  not  for  technology 
but  for  making  individualized,  tutorial  instruction  affordable,  thereby  substantially  increasing  the 
efficiency  of  education  and  training  and  our  ability  to  accelerate  acquisition  of  knowledge,  skill,  and 
expertise. 

•  The  instructional  technologies  targeted  by  ADL  have  been  found  to  be  more  effective  than  typical 
classroom  instruction  across  many  instructional  objectives  and  subject  matters. 

•  ADL  is  generally  less  costly,  offering  greater  Return  On  Investment  (ROI)  than  current  instructional 
approaches,  especially  when  many  widely  dispersed  students  must  be  served. 

•  ADL  allows  education,  training,  performance  /  decision  aiding,  and  problem  solving  to  be  delivered 
from  the  same  knowledge  bases  on  platforms  ranging  from  hand-held  devices,  to  large  desk-top 
computers,  to  capabilities  embedded  in  operational  equipment. 

•  This  RTG  served  as  the  vehicle  to  maintain  ADL  STO  active  and  productive  within  the  NATO 
environment.  ADL  has  succeeded  far  beyond  its  expectations  and  much  of  its  success  could  be 
attributed  to  two  closely  associated  NATO  activities.  RTG  HFM-212  that  served  as  the  S&T  focus  for 
NATO  and  the  NATO  Training  Group’s  working  group  on  Individual  Training  and  Educational 
Development  that  served  as  the  schoolhouse  application  for  NATO  commands.  The  many  uniformed 
and  civilian  government  professionals  who  were  active  in  both  these  organizations  are  the  strength  of 
ADL’s  operational  research  capabilities. 
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2.0  OBJECTIVES 

The  Human  Factors  and  Medical  (HFM)  Panel’s  research  technology  effort  for  ADL  has  three  main  objectives: 

1)  To  identify  and  explore  operational  needs  and  technology  capabilities  for  an  agile,  open  ADL 
framework  for  content  and  infrastructure  that  can  be  leveraged  for  global  collaboration  across  NATO, 
PfP,  MD,  and  other  coalition  partners.  This  objective  became  one  of  the  main  discussion  points  for 
Coalition  Warfare  Program  participants  as  well  as  RTG  HFM-212.  Consensus  was  varied  due  to  the 
many  differences  in  each  Nation’s  own  programs.  It  was  here  that  multi-domain  collaboration  became  a 
valuable  asset. 

2)  To  actively  explore  the  continued  effectiveness  of  the  ADL  approach  and  its  capabilities  to  assess  and 
track  learner  success  while  utilizing  advanced  distributed  learning  technologies  (i.e.,  conventional, 
traditional  and  emerging). 

3)  To  deliver  a  meaningful  final  technical  report  that  each  Nation  can  use  in  pursuing  its  own  development 
and  use  of  distributed  learning  technologies. 

The  NATO  Training  Group’s  Working  Group  on  Individual  Training  and  Educational  Development  (IT&ED) 
has  been  a  major  partner  and  contributor  by  helping  transition  the  RTG  work  to  in-country  schoolhouse 
applications,  and  even  today  it  continues  to  identify  and  assess  how  to  best  implement  advances  in  ADL  for 
NATO  training. 

Annex  A  of  this  report  is  a  summary  of  research  underlying  ADL.  It  was  compiled  by  one  of  the  USA  RTG 
HFM-212  delegates,  Dr.  J.D.  Fletcher,  a  distinguished  scientist  at  the  Institute  for  Defense  Analyses. 

3.0  MEMBERSHIP  IN  THE  RTG 

Membership  in  RTG  HFM-212  is  as  follows.  Unofficial  delegates  from  the  NATO  Training  Group  also  greatly 
contributed  to  the  findings  and  transition  of  this  technology  to  the  Alliance: 

•  Canada; 

•  France; 

•  Finland; 

•  Italy; 

•  Netherlands; 

•  Norway; 

•  Sweden; 

•  Switzerland; 

•  Ukraine; 

•  United  Kingdom;  and 

•  United  States. 

Leadership  for  RTG  HFM-212  was  provided  by  its  Co-Chairs:  Dr.  Ray  Perez  and  Dr.  Kristy  Murray  from  the 
United  States. 
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Dr.  Murray  was  also  a  member  of  the  NATO  NTG’s  1T&ED  mentioned  earlier.  This  dual  NATO  membership 
greatly  enhanced  the  RTG’s  ability  to  stay  relevant  to  the  NATO  via  the  ACT  and  therefore  the  NATO  training 
and  education  users. 

The  NATO  HFM  RTG  Mentor  was  Mr.  Paul  Chatelier,  United  States  and  the  Lead  Nation  for  RTG  HFM-212 
was  the  United  States. 


4.0  PARTICIPATION  BY  PARTNER  NATIONS 

There  was  broad  participation  by  Partner  Nations:  Open  to  all  PfP,  MD  and  Selected  Contact  Countries 
(Australia,  Japan,  New  Zealand  and  South  Korea).  Interest  and  some  participation  from  PfP  Nations 
included  Azerbaijan,  Finland,  Moldova,  Romania  and  Sweden. 


5.0  MEETINGS  HELD  BY  RTG  HFM-212 

•  First  meeting  took  place  in  Venice,  Italy  June  15-16,  2010. 

The  agenda  consisted  of  review,  discussion,  and  approval  of  the  Technical  Activity  Plan  (TAP)  and  the 
Terms  of  Reference  (ToR)  for  the  proposed  HFM-212  RTG. 

Dr.  Perez  the  co-chairman  of  the  RTG  presented  for  discussion  a  list  of  research  topics  to  be  pursued  by  the 
RTG.  These  are: 

•  Adaptive  Instruction  (Focused  on  intelligent  tutoring  features). 

•  Adaptive  Assessment  (a  much  richer  assessment  environment  than  currently  found  in  applications 
that  now  conform  to  ADL’s  (Shared  Content  Object  Reference  Model  (SCORM);  Blended  Learning/ 
Intelligent  Learning  Technologies,  which  combine  learning  experiences  from  e-leaming  and 
classroom  learning  with  data  from  each  medium  made  accessible  to  each  form  of  instruction). 

•  Learner  Data  (a  rich  set  of  data  that  describes  the  learner,  used  as  a  basis  for  adaptive  learning). 

•  Collaboration  Environments  for  both  instructors  and  students  to  collaborate  with  their  peers. 

•  Read-Write  Learning  Management  System  (LMS)  (enables  instructors  to  retrieve  data  from  an 
LMS  and  to  record  data  in  the  LMS  from  classroom-based  instruction  and  from  other  Computer- 
Based  Training  (CBT)  not  controlled  by  the  LMS). 

•  Data  Persistence/mining  (data  from  various  experiences  can  be  stored,  shared,  and  used  in  a  manner 
enabling  a  student/trainee  to  leave  the  environment  and  return  exactly  where  he/she  left  off). 

•  Security/authentication  (especially  important  for  business  models  and  privacy  of  trainees). 

•  Alternative  devices/formats  (especially  for  hand-held  devices). 

•  Sequencing  logic  extensions  available  to  SCORM-conformant  courseware  by  providing  a 
mechanism  for  plug-and-play  distributed  learning  environments. 

•  Extend  support  for  simulation-based  training  by  formally  adopting  High  -Level  Architecture  (HLA) 
and  other  modeling  and  simulation  protocols  to  consistently  allow  the  sharing  of  performance- 
related  training  data. 

•  Continue  investigations  to  optimize  the  co-management  of  training  and  technical  data  using 
standards  like  S1000D. 
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•  Investigate  how  (e.g.,  directive,  reflective,  explicit,  implicit)  and  when  (e.g.,  real  time,  after  action 
review)  to  deliver  feedback  that  will  maximize  learning  from  intelligent  tutor-enhanced  serious 
games.  As  discussions  concerning  learning  from  immersive  serious  games  began  to  deal  with 
learning,  they  led  to  investigations  of  the  extent  to  which  stories  and/or  competition  enhances 
learning. 

RTG  members  agreed  that  these  were  important  topics  deserving  serious  attention  and  they  would  serve  as 
the  topics  to  be  covered  in  future  RTG  HFM-212  meetings.  Unfortunately,  the  time  and  resources  available 
to  the  RTG  members  often  limited  thorough  investigation  of  these  topics. 

•  RTG  HFM-212’s  Spring  2012  meeting  was  held  in  conjunction  with  the  International  Training  and  Education 
Conference  (ITEC)  2012  on  May  24,  2012  in  London,  England.  Taking  advantage  of  conference  facilities, 
they  met  jointly  with  the  NATO  Training  Group  (IT&ED)  working  group  on  Individual  Training  and  RTG 
UFM-212.  This  meeting  identified  mobile  medical  uses  and  opportunities  for  joint  programs. 

•  The  Spring  2013  meeting  was  held  on  26  February  2013  in  the  USA  at  the  ADL  Laboratory  in  Orlando,  FL, 
USA.  Participants  included:  Dr.  Ray  Perez,  USA;  Dr.  Kristy  Murray,  USA;  Mr.  Paul  Chatelier,  USA; 
Commander  Geir  Isaksen,  NOR;  LtCol  Rolf  Thielmann,  DEU:  Mr.  Bill  Railer  and  Lieutenant-Commander 
Remi  Tremblay,  CAN.  This  meeting  focused  on  a  mobile  project  jointly  developed  with  Canada,  Germany, 
Norway,  Poland  and  the  United  States.  The  proof  of  concept  utilized  the  experience  API  (developed  by  the 
ADL  Initiative)  to  deliver  and  track  content  via  a  mobile  device. 

•  The  Fall  2013  meeting  was  held  in  conjunction  with  the  Interservice/Industry  Training,  Simulation, 
and  Education  Conference  (I/ITSEC)  in  Orlando,  FL,  USA,  during  December  2013.  This  meeting  identified 
and  coordinated  demonstrations  including  timelines  and  national  responsibilities.  It  also  determined 
opportunities  for  lectures  on  advances  in  ADL  for  mobile  learning  and  began  to  place  added  emphasis  on 
mobile  medical.  Due  to  the  success  of  this  meeting,  where  many  members  would  be  meeting  anyway, 
the  attendees  decided  to  hold  meetings  of  opportunity  with  NATO  attendees  at  subsequent  I/ITSEC 
conferences. 


6.0  RTG  HFM-212  DELIVERABLES 

It  should  be  emphasized  that  these  deliverable  are  the  result  of  the  joint,  collaborative  nature  of  the  RTG. 
Since  the  overall  goal  was  collaboration  and  sharing  of  resources  and  opportunities,  they  were  focused  on 
transition  to  the  potential  population  and  emphasized  shared  deliverables  and  knowledge. 

1)  RTG  HFM-212  identified  training  technology  requirements  and  challenges  associated  with  the 
open  ADL  framework: 

•  Improving  sustainment  training  to  include  just-in-time/on-demand  training; 

•  Conducting  web-based  team  training  (to  include  mission  rehearsal); 

•  Locating  clear  guidance  from  leadership  on  Framework  Standards; 

•  Performing  Retum-on-Investment  analysis; 

•  Developing  common  tools  and  defined  processes  to  assist  media  selection; 

•  Overcoming  Bandwidth  issues; 

•  Improving  synthetic  training  to  permit  more  dynamic  presentation  of  training  material  (spanning  the 
current  gap  between  classroom  and  live  training); 
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•  Improving  real-world  training  through  simulations  and  virtual  environments  to  include  assessments, 
tracking,  and  reporting  in  these  environments;  and 

•  Upgrading  legacy  content  and  systems. 

2)  Training  and  Learning  Architecture  -  The  ADL  Initiative  (USA)  is  developing  an  open  source 
framework  called  the  Training  and  Learning  Architecture  (TLA).  The  TLA  is  an  umbrella  term 
encompassing  all  technologies  designed  to  create  a  rich  environment  for  online  training  and  learning. 
A  goal  of  the  TLA  is  to  ensure  platform  neutrality,  which  will  accommodate  intermittent  or  disconnected 
network  scenarios,  and  the  capability  to  move  learning  out  of  the  desk-top  browser.  The  four  pillars  of 
the  TLA  will  include  Experience  Tracking,  Learner  Profiles,  Content  Brokering,  and  Competency 
Networks.  At  the  time  of  this  report  only  the  experience  tracking  pillar  is  developed  to  the  extent  that  it 
is  a  usable  capability. 

•  Experience  Tracking  -  The  Experience  API  (xAPI)  is  a  suite  of  web  services  that  allows  storage 
and  reporting  of  learning  experiences.  Experience  tracking  can  be  applied  to  formal,  classroom, 
eLearning,  game,  and  virtual  environments,  as  well  as  social  and  other  informal  learning  experiences. 
The  Learning  Record  Store  (LRS)  is  the  experience  tracking  and  storage  component  of  ADL’s 
service-based  approach  to  the  TLA.  The  platform-independent  LRS  design  allows  flexibility  in  that 
it  may  be  a  stand-alone  service  or  a  complementary  component  of  a  traditional  LMS.  In  addition, 
the  LRS  allows  authorized  systems  to  retrieve  previously  recorded  statements,  which  enable  the 
development  of  advanced  third-party  reporting  and  data  analytics  tools.  The  xAPI  also  moves 
beyond  the  single  learner  approach,  allowing  for  team-based  exercises,  collaboration,  and  direct 
instructor  intervention.  This  capability  enables  group  learning,  formal  and  informal  learning,  and 
social  learning  (on  any  device  or  platform).  Any  system  that  “understands”  the  xAPI  can  share  data. 
Examples  include  assessment  systems,  HR  systems,  gradebook  applications,  statistical  and 
reporting  systems,  data  bases  being  assembled  for  data  mining,  etc. 

3)  Mobile  Learning  +  xAPI  Research  Project  -  Naval  Post-Graduate  School  (USA)  Project  (2013  - 
2014)  explores  the  potential  of  using  ADL’s  Experience  API  (xAPI)  to  track  content  delivered  outside 
the  Navy’s  e-Leaming  (NeL)  Learning  Management  System  (LMS)  and  to  synchronize  recorded  data 
with  the  LMS.  This  work  will  include  identification  of  metrics  that  can  be  compiled  and  used  beyond 
basic  course  completion  data. 

4)  Mobile  Learning  Environment  (MoLE)  October  2010  to  October  2012: 

•  MoLE  is  a  Coalition  Warfare  Program  (CWP)  funded,  Office  of  Naval  Research  London  (ONRL) 
grant  to  create  a  reference  architecture  that  enables  Joint  Knowledge  Online  (JKO)  to  support 
effective  mobile  learning  content. 

•  Tribal,  Inc.  (a  British  contractor)  was  responsible  for  active  collaboration  with  JKO.  Through  2012 
Tribal  assisted  with  the  ongoing  transition  to  further  define  and  develop  the  m-Leaming  Suite  in 
accordance  with  JKO  requirements.  Unfortunately  funding  from  CWP  was  not  continued  so  the 
contractor  is  no  longer  working  on  this  program. 

•  Design  and  develop  technical  solutions  to  deliver  mobile  learning  content  within  DoD  constraints. 

•  A  final  solution  incorporating  the  MoLE  app  (for  both  Android  and  iOS)  and  the  “data  collection” 
back-end  portal/services  were  delivered  for  Proof  Of  Concept  (POC).  Post-POC,  further  developments 
were  implemented  to  the  app  regarding  improved  “Network”  functionality.  Pinal  versions  of  the 
apps  are  now  live  in  the  relevant  stores. 
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•  Twenty-four  Partner  Nations  participated  in  this  2-year  effort. 

•  The  m-Leaming  Suite  (vl)  went  live  during  August  2012  within  the  JKO  architecture. 
It  demonstrated  successful  deployment,  app  store  approval,  multi-device  installations,  multi-lingual 
usage,  broad  user  base.  All  MoLE  mobile  content  is  available  to  key  transition  partners  in  converted 
format,  either  as  stand-alone  media  (eBooks,  video),  as  mobile  courses  (e.g.,  Combating  Trafficking 
in  Persons  (CTiP)),  as  modules  within  the  JKO  app,  or  as  free  access  apps. 

•  The  CWP  MoLE  project  transitioned  into  two  applications: 

•  The  Global  MedAid  App  -  Enabling  worldwide  medical  use,  including  the  civilian  medical 
community;  and 

•  The  JKO/ ADL  Mobile  Hybrid  App  -  Covering  joint  and  coalition  military  training. 

5)  JKO/ ADL  (Joint  Knowledge  on  Line/ADL)  Mobile  Project  -  One  MoLE  transition  has  become 
JKO/ADL  Mobile,  which  is  jointly  sponsored  by  JKO  and  the  ADL  Initiative,  but  also  includes  Canada, 
Great  Britain,  Poland  and  NATO  ACT. 

Features  of  JKO/ADL  Mobile  are  as  follows: 

•  Instructional  strategies  for  mobile  learning  strategies; 

•  Guidance  and  best  practices  on  mobile  learning  development; 

•  Free  App  available  in  Apple  and  Android  App  stores; 

•  Download  and  complete  courses  on  mobile  device  (tablets  and  phones); 

•  Course  completions  synchronized  with  JKO  desk-top  system  for  tracking  and  reporting; 

•  Downloadable  job  aids; 

•  The  latest  JKO/ADL  news; 

•  Access  to  podcasts,  videos,  and  e-books; 

•  Course  content  segregated  by  participating  partners  and  PIN;  and 

•  Public  content  and  news  announcements  available  without  PIN. 

6)  The  xAPI  International  Mobile  Pilot  (xAMP)  -  Phase  I  was  completed  by  a  Norway/US  cooperative 
project  in  2013.  xAMP  Phase  I  included  the  creation  of  an  xAPI-enabled  mobile  learning  experience 
(Rules  of  Engagement  introduction  module),  deployment  of  US  and  Norway  Learning  Record  Stores 
(LRS),  and  the  determination  and  implementation  of  basic  tracking,  reporting,  and  analytic  requirements. 

•  Phase  II  of  xAMP  includes  the  following  potential  requirements: 

•  Deploy  additional  LRSs  in  participating  countries; 

•  Identify  a  multi-national  end-user  group  for  pilot  testing; 

•  Perform  a  pilot  test  with  the  user  group  including  a  questionnaire  and  xAPI  data  collection; 

•  Create  analytic  and  reporting  tools  to  process  data  collected  in  the  pilot  test;  and 

•  Publish  findings. 

•  Other  potential  requirements  include: 

•  Modification  of  the  Rules  of  Engagement  content  to  a  native  mobile  app  (from  HTML5); 
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•  Creation  of  additional  Rules  of  Engagement  modules  for  inclusion  in  the  pilot  test;  and 

•  LRS  synchronization  functions  to  share  data  between  LRSs. 

7)  Re-Usability  Support  System  for  E-Learning  (RUSSEL)  -  The  goal  of  the  RUSSEL  project  was  to 
improve  efficiency  in  content  development  through  the  reuse  of  existing  learning  materials.  RUSSEL  is 
an  out-of-the-box  repository  and  digital  library.  Instructional  design  is  achieved  through  a  built-in 
Electronic  Performance  Support  System  (EPSS)  that  is  facilitated  by  easy-to-find/use  courseware, 
content,  media  files,  and  SCORM  assets.  It  is  completely  open  source  (https://github.com/adlnet/ 
RUSSEL). 

8)  The  Multi-Channel  Learning  System  (MLS)  Project  -  Although,  not  a  direct  outcome  of  the  RTG, 
MLS  was  stimulated  by  discussion  among  members  of  the  RTG  and  identified  by  the  RTG  as  having 
potential  for  large-scale  usage  by  NATO  and  Partner  Nations.  The  MLS  project  was  proposed  jointly  by 
Naval  Education  and  Training  Security  Assistance  Field  Activity  (NETSAFA)  and  the  Defense  Institute 
of  Security  Assistance  Management  (DISAM).  The  effort  focused  on  identifying  best  methods  for 
providing  distance  education  for  international  military  students  preparing  for  resident  training  in  the 
United  States.  The  goals  of  the  MLS  Project  were  to: 

1)  Ensure  that  International  Military  Student  Pre-Departure  Brief  (IMSPDB)  students  have  a 
positive  and  successful  experience  in  the  US  along  with  their  US-counterparts; 

2)  Identify  the  best  methods  for  providing  distance  education  for  international  military  students; 
and 

3)  Evaluate  the  effectiveness  of  using  multiple  learning  fonnats  to  support  Security  Cooperation 
Education  and  Training  Program  (SCETP)  requirements. 

However,  the  project  was  not  funded. 


7.0  CONCLUSIONS  AND  SUMMARY 

RTG  HFM-212  enjoyed  continual  success  in  reaching  its  stated  objectives  as  listed  in  the  TAD/TAP.  It  must  be 
emphasized  that  this  success  was  due  to  a  substantial  NATO/PDP  organizational  and  policy  base  as  well  as  a 
strong  working  relationship  with  the  NATO  Allied  Command  Transformation  (ACT)  in  Norfolk,  VA,  USA. 
With  the  funding  support  provided  by  the  CWP  (Coalition  Warfare  Program),  the  excellent  collaboration  and 
data  sharing  of  the  MoLE,  and  joint  funding  from  the  US  Army  Telemedicine  and  Advanced  Technology 
Research  Center  (TATRC),  RTG  HFM-212  is  considered  an  outstanding  success.  It  utilized  an  existing 
NATO  infrastructure,  actively  collaborated  with  other  NATO  assets  (e.g.,  the  NATO  Training  Group  and  the 
PfP  Consortium),  and  built  a  transition  path  toward  the  delivery  of  its  work  to  NATO  schoolhouses. 
By  accomplishing  the  RTG  goals  of  creating  and  strengthening  partnerships  among  24  key  Partner  Nations, 
RTG  HFM-212  has  contributed  sizable  technology  awareness  as  well  as  real  capabilities  to  the  NATO  education 
and  training  communities. 

A  number  of  important  projects  were  motivated  by  the  efforts  of  this  RTG.  These  projects  were  described  in 
Section  6.0.  These  projects  have  the  potential  for  large-scale  use  by  NATO  and  Partner  Nations.  The  way  ahead 
for  research  in  this  area  is  reflected  in  the  proposed,  but  unfunded,  MLS  Project.  Where  this  project  focused  on 
identifying  the  best  methods  for  providing  distance  education  for  international  military  students  preparing  for 
resident  training  in  the  US,  its  transition  to  the  NTG  IT&ED  allows  it  to  continue  contributing  to  NATO 
education  and  training  schoolhouses.  Through  this  broad  collaboration  within  the  alliance,  RTG  HFM-212  was 
able  to: 
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1)  Develop  a  capability  that  ensures  International  Military  Student  Pre-Departure  Brief  (1MSPDB)  students 
have  a  positive  and  successful  experience  in  the  US  along  with  their  US-counterparts; 

2)  Identify  the  best  methods  for  providing  distance  education  for  international  military  students;  and 

3)  Evaluate  the  effectiveness  of  multiple  learning  formats  to  support  Security  Cooperation  Education  and 
Training  Program  (SCETP)  requirements. 

Finally,  RTG  EIFM-212  initiated  application  of  ADL  capabilities  in  the  medical  and  health  sciences  community 
via  its  sponsor,  the  NATO  Eluman  Factors  and  Medical  Panel. 
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A.O  OVERVIEW 

The  Advanced  Distributed  Learning  (ADL)  Initiative  was  established  by  the  Office  of  the  Undersecretary  of 
Defense  for  Personnel  and  Readiness  (OUSD(P&R))  in  1997.  Its  purpose  was  to  assist  the  military  services  in 
making  “learning”  (education,  training,  and  performance  /  decision  aiding)  available  on-demand,  anytime  and 
anywhere. 

“Learning”  in  ADL  is  used  as  a  catch-all  designator  for  education,  training,  and  performance/decision  aiding. 
“Distributed”  in  ADL  signifies  learning  that  can  be  provided  in  classrooms  with  a  teacher  present,  in  the  field 
linking  together  widely  dispersed  instructors  and  students,  or  standing  alone  with  no  instructor  other  than  the 
computer  itself.  “Advanced”  in  ADL  implies  affordable,  interactive,  adaptive,  on-demand  instruction  delivered 
using  computer  technology  so  that  it  is  available  anytime,  anywhere. 

Empirical  research,  available  as  early  as  the  1960s,  suggested  the  feasibility  of  ADL  and  its  goals.  It  has  shown 
that: 


•  Individualized,  tutorial  Teaming’  (including  individualized  performance  /  decision  aiding)  can  be 
provided  affordably  by  technology-based  learning. 

•  Technology-based  learning  can  be  more  effective  and  can  produce  greater  return  on  investment  than 
conventional  instructional  approaches  across  many  instructional  objectives  and  subject  matters. 

•  Technology-based  learning  allows  education,  training,  performance  aiding,  and  decision  aiding  to 
be  delivered  on  platforms  ranging  from  hand-held  devices,  to  desk-top  computers,  to  capabilities 
embedded  in  operational  equipment. 

Statistical  findings  from  this  research  may  be  summarized  by  a  “Rule  of  Thirds.”  It  states  that  application  of 
technology-based  learning  can  reduce  the  cost  of  instruction  by  about  one-third.  Additionally,  it  can  either 
reduce  instructional  time  to  reach  instructional  goals  by  about  one-third,  or  increase  the  skills  and  knowledge 
acquired  by  about  one-third  while  holding  instructional  time  constant. 

The  long-term  vision  for  ADL  is  an  extrapolation  from  such  developments  as  portable,  increasingly  accessible 
computing,  the  global  information  infrastructure,  modular  object-oriented  architectures,  and  natural  language 
processing.  The  march  toward  devices  that  might  be  described  as  personal  learning  associates  seems  inevitable. 
These  devices  will  act  as  personal  accessories.  They  will  respond  on  demand  to  requests  for  education,  training, 
and  performance  aiding  by  assembling  relevant  objects  from  the  global  infrastructure  and  engaging  the  user  in 
guided  conversations  to  enhance  the  knowledge  and  skills  and/or  problem-solving  capabilities  of  individuals 
and/or  groups  of  dispersed  individuals  whose  devices  are  wirelessly  linked  together. 


A.l  THE  REQUIREMENT 

About  1.1  million  US  forces  are  dispersed  across  the  Continental  United  States  and  an  additional  1.4  million 
forces  are  spread  across  50  foreign  countries.  Individuals  in  these  forces  need  to  be  capable  of  independent 
thought  and  action.  They  also  need  to  be  able  to  continue  their  training  and  career  growth  when  they  are  far 
removed  from  military  schoolhouses,  expert  mentors,  or  others  in  their  own  career  specialties. 
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Additional  complications  arise  from  the  rapid  growth  in  the  amount  and  complexity  of  information  that 
individuals  at  all  levels  must  integrate  and  prioritize.  The  trend  has  produced  an  increasing  demand  for 
what  Wulfeck  and  Wetzel-Smith  (2008)  [51]  and  Wetzel-Smith  and  Wulfeck  (2010)  [49]  have  described  as 
“incredibly  complex  tasks.”  Today,  about  15%  of  military  tasks  are  abstract,  multi-dimensional,  non-linear, 
dynamic,  and  inter-dependent.  The  dynamic  nature  of  these  tasks  and  the  evolving  operational  environment 
require  that  individuals  receive  up-to-date  training  and  performance  assistance. 

Finally,  the  difficulty  of  all  military  tasks  is  exacerbated  by  the  dispersal  of  units  and  the  individuals  serving 
within  these  units.  Human  decisions  and  actions  must  be  coordinated  within  and  across  teams  whose  members 
may  be  globally  dispersed  in  other  Nations,  immersed  in  other  cultures,  and  serving  at  widely  varied  command 
levels.  The  ability  to  communicate,  coordinate,  and  perform  tasks  under  these  conditions  may  not  guarantee 
success,  but  the  consequences  of  its  absence  are  severe. 

These  operational  conditions  have  created  an  imperative  to  ensure  that  “learning”  -  education,  training,  and  task, 
job,  and  decision  aiding  -  is  rapidly  available  on  demand,  anytime  and  anywhere,  to  unifonned  and  civilian 
individuals  and  teams  at  all  levels  of  responsibility. 


A.2  THE  TECHNICAL  OPPORTUNITY:  A  THIRD  REVOLUTION  IN  LEARNING 

In  response  to  this  requirement,  ADL  is  riding  and  contributing  to  a  third  revolutionary  wave  in  learning. 
The  first  revolution  in  learning  occurred  about  5,000  years  ago,  with  the  invention  of  writing  -  the  use  of  graphic 
tokens  to  represent  syllables  of  sound.  Before  the  invention  of  writing,  learning  activity  appears  to  have  been 
conducted  as  a  tutorial  conversation  between  a  learner  and  a  sage,  or  at  least  someone  who  had  the  knowledge 
and  skills  the  learner  sought  to  acquire.  With  writing,  a  learner  could  access  knowledge  and  skill  without  this 
face-to-face  interaction.  Writing  allowed  the  content  of  ideas  and  instruction  to  transcend  time  and  place. 

The  second  revolution  in  learning  occurred  with  the  invention  of  books  printed  from  moveable  type  -  first  in 
China  around  1000  A.D.  and  then  in  Europe  in  the  mid- 1400s  [31].  With  books,  the  dissemination  of  knowledge 
and  skills  through  writing  became  scalable.  Once  content  was  produced,  it  could  be  made  widely  available 
and  became  increasingly  inexpensive  as  printing  technology  developed.  However,  with  writing  and  printing, 
the  dissemination  of  content  was  passive.  It  lacked  the  tutorial  interactivity  that  had  been  the  foundation  of 
learning  for  the  previous  100,000  years  or  so  of  human  existence. 

Enter  computer  technology,  with  its  ability  to  adapt  rapidly,  in  real  time,  to  the  changing  demands,  needs, 
and  circumstances  of  learners  and  learning.  Computer  technology  allows  not  just  content,  but  also  instructional 
strategies,  techniques,  and  interactions  to  become  inexpensively  ubiquitous  and  available  on  demand  -  anytime, 
anywhere.  It  may  be  fomenting  a  third  revolution  in  learning.  ADL  is  a  response  and  a  contributor  to  this  third 
revolutionary  possibility. 

“Learning”  in  ADL  is  used  as  a  catch-all  designator  for  education,  training,  and  performance  /  decision  aiding. 
“Distributed”  in  ADL  is  not  just  another  word  for  distance.  It  signifies  learning  that  can  be  provided  in 
classrooms  with  a  teacher  present,  in  the  field  linking  together  widely  dispersed  instructors  and  students, 
or  standing  alone  with  no  instructor  other  than  the  computer  itself.  “Advanced”  in  ADL  implies  affordable, 
interactive,  adaptive,  on-demand  instruction  using  computer  technology  so  that  it  can  be  delivered  anytime, 
anywhere. 

The  ADL  purpose,  from  its  inception,  has  been  to  ensure  access  to  the  highest  quality  education,  training,  and 
performance/decision  aiding  tailored  to  individual  needs  and  delivered  cost  effectively  and  anytime/anywhere. 
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A.3  EVIDENCE :  RE  SEARCH  AND  DEVELOPMENT  (R&D)  FOUNDATIONS 

What  evidence  suggests  that  computer  technology  might  be  affecting  this  third  revolution?  What  have  we 
learned  from  research  on  computer  uses  in  instruction?  Some  key  findings  can  be  summarized  as  follows: 

•  Although  individualized  learning  tailored  to  the  needs  of  individual  students  has  long  been  viewed  as  an 
imperative,  it  has  also  been  viewed  as  unaffordable  [40].  With  few  exceptions,  we  cannot  afford  one 
instructor  for  every  student  -  an  Aristotle  for  every  Alexander.  Computer  technology  can  make 
this  imperative  affordable.  A  core  argument  for  ADL,  then,  is  not  for  technology  but  for  making 
individualization  affordable. 

•  The  instructional  technologies  targeted  by  ADL  have  been  found  to  be  more  effective  than  typical 
classroom  instruction  across  many  instructional  objectives  and  subject  matters. 

•  ADL  is  generally  less  costly,  offering  greater  Return  On  Investment  (ROl)  than  current  instructional 
approaches,  especially  when  many  widely  dispersed  students  must  be  served. 

•  ADL  allows  education,  training,  and  performance  /  decision  aiding  and  problem  solving  to  be  delivered 
from  the  same  knowledge  bases  on  platforms  ranging  from  hand-held  devices,  to  large  desk-top 
computers,  to  capabilities  embedded  in  operational  equipment. 

These  arguments  have  been  made  for  the  computer-assisted  approaches  used  by  ADL  for  the  last  40  -  50  years 
(e.g.,  [1],  [3],  [10],  [23],  and  [42]).  They  have  been  repeatedly  validated  by  empirical  research  and  practical 
experience.  Statistical  findings  from  this  work  have  been  summarized  by  a  “Rule  of  Thirds”  [15],  [16].  This  rule 
states  that  application  of  the  technologies  on  which  ADL  is  based  reduces  the  cost  of  instruction  by  about 
one -third.  In  addition,  the  application  of  these  technologies  can  either  reduce  instructional  time  to  reach 
instructional  goals  by  about  one-third  or  increase  the  skills  and  knowledge  acquired  by  about  one-third  while 
holding  instructional  time  constant. 

The  following  sections  discuss  more  specifically  the  R&D  behind  these  arguments  and  the  Rule  of  Thirds. 


A.4  INDIVIDUALIZATION:  TUTORIAL  INSTRUCTION 

The  argument  for  ADL  technology  begins  with  an  issue  that  arises  independently  from  applications  of 
technology.  It  concerns  the  effectiveness  of  classroom  instruction,  involving  one  instructor  for  20  -  30  (or  more) 
students,  compared  to  individual  tutoring,  involving  one  instructor  for  each  student.  Empirical  results  from 
comparisons  of  this  sort  are  shown  in  Figure  A-l,  adapted  from  Bloom  (1984)  [4]. 
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Achievement 


Figure  A-1:  Individual  Tutoring  Compared  to  Classroom  Instruction. 


Bloom  combined  findings  from  three  empirical  studies  that  compared  tutoring  with  one-on-many  classroom 
instruction.  The  result  of  such  comparisons  showed  the  tutored  students  to  have  learned  more,  and  the  result  is 
not  surprising.  The  surprise  is  the  size  of  the  difference.  Overall,  as  Figure  A-1  suggests,  the  difference  was 
found  to  be  two  standard  deviations.  It  suggests  that  one-on-one  tutoring,  with  instructional  time  held  constant, 
can  raise  the  performance  of  mid-level  50th-percentile  students  roughly  to  that  of  98th-percentile  students.  These 
and  similar  empirical  findings  suggest  that  differences  between  one-on-one  tutoring  and  typical  classroom 
instruction  are  not  only  likely  but  very  large. 

Most  importantly  for  training  applications,  the  shapes  of  these  distributions  support  Como  and  Snow’s  (1986) 
[9]  suggestion  that  the  individualization  provided  by  tutorial  instruction  helps  guarantee  that  all  learners  reach 
some  basic  level  of  competency. 

What  accounts  for  the  success  of  one-on-one  tutoring?  The  research  summarized  below  suggests  that  it  is 
primarily  due  to: 

1 )  The  ability  of  tutors  and  their  students  to  engage  in  many  more  interactions  per  unit  of  time  than  is 
possible  in  a  classroom;  and 

2)  The  ability  of  tutors  to  tailor  pace,  sequencing,  and  content  to  the  needs,  capabilities,  goals,  interests, 
and  values  of  individual  students. 

A.4.1  Interactivity 

With  regard  to  the  first  tutorial  capability  (the  intensity  of  instructional  interaction),  Graesser  and  Person  (1994) 
[27]  reported  the  following: 

•  Average  number  of  questions  by  a  teacher  of  a  class  in  a  classroom  hour:  3. 

•  Average  number  of  questions  asked  by  a  tutor  and  answered  by  a  student  during  a  tutorial  hour: 
120-  145. 

•  Average  number  of  questions  asked  by  any  one  student  during  a  classroom  hour:  0.11. 

•  Average  number  of  questions  asked  by  a  student  and  answered  by  a  tutor  during  a  tutorial  hour: 
20-30. 
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These  data  show  great  differences  in  interactivity  and  intensity  between  tutorial  and  classroom  instruction. 
This  level  of  interactivity,  by  itself,  may  account  for  a  substantial  portion  of  the  success  of  tutorial  over 
classroom  instruction. 

Is  this  level  of  interactivity  found  in  instruction  using  ADL  technology?  Early  studies  of  computer-assisted 
instruction  in  reading  and  arithmetic  found  that  students  in  grades  K-6  were  answering  8-10  individually 
selected  and  assessed  questions  each  minute  [17],  [19].  This  level  of  interactivity  extrapolates  to  480  -  600 
such  questions  an  hour  if  students  were  to  sustain  this  level  of  interaction  for  60  minutes. 

A.4.2  Tailoring  Pace,  Sequencing,  and  Content 

With  regard  to  the  second  tutorial  capability  (tailoring  the  session  content),  tutors  adjust  the  content  and 
sequence  of  instruction  to  the  needs  of  their  students.  All  these  adjustments  relate  to  pace  -  the  rate  or  speed  with 
which  students  are  allowed  to  proceed  through  instructional  material. 

Many  classroom  instructors  have  been  struck  by  the  differences  in  the  pace  with  which  their  students  leam. 
Their  observations  are  confirmed  by  research.  For  instance,  consider  some  findings  on  the  time  it  takes  for 
different  students  to  reach  the  same  instructional  objectives: 

•  Ratio  of  time  needed  by  fastest  and  slowest  students  to  reach  mathematics  objectives:  4  to  1  [43],  [44]. 

•  Overall  ratio  of  time  needed  by  fastest  10%  and  slowest  10%  of  K-8  students  to  reach  objectives  in  a 
variety  of  subjects:  5  to  1  [25]. 

•  Ratio  of  time  needed  by  fastest  and  slowest  undergraduates  in  a  major  research  university  to  leam  a 
programming  language:  7  to  1  (Private  communication,  Corbett,  1998). 

The  differences  in  the  speed  with  which  students  leam  are  not  surprising,  but,  as  with  tutoring,  the  magnitudes  of 
the  differences  are  surprising.  Although  the  speed  with  which  different  students  reach  instructional  objectives  is 
not  independent  of  ability,  research  has  found  that  it  is  most  directly  keyed  to  prior  knowledge  [11],  [47]. 
Students  in  military  education  and  training  bring  with  them  a  wide  variety  of  backgrounds  and  life  experiences  - 
often  much  wider  than  those  found  among  K-12  students.  Adjusting  the  pace  of  instmction  to  their  individual 
needs  may  be  especially  important  for  them. 

The  challenge  this  diversity  presents  to  classroom  instmctors  is  daunting.  Typically,  the  instmctors  focus  on 
some  of  their  students  and  leave  the  others  to  fend  for  themselves.  This  pattern  is  especially  true  in  training 
settings  where  the  primary  task  is  to  enable  as  many  learners  as  possible  to  cross  a  specific  threshold  of 
knowledge  and  skill.  Technology  alleviates  this  difficulty  because  it  allows  each  learner  to  proceed  as  rapidly  or 
as  slowly  as  needed.  Learners  can  skip  what  they  already  know  and  concentrate  on  what  they  need  to  leam. 

The  degree  to  which  individualization  of  sequence  and  content  matters  is  to  some  extent  addressed  by  studies 
comparing  individualized  branching  with  fixed-content,  linear  sequencing.  Two  of  the  early  studies  were 
performed  by  Fowler  (1980)  [21]  and  Verano  (1987)  [48]. 

Both  of  these  researchers  used  computer-controlled  videodisc  instmction  in  their  experiments.  Fowler  compared 
branched  presentations  with  linear  instmction  in  which  precisely  the  same  materials  were  held  to  a  fixed-content, 
linear  sequence.  She  reported  an  effect  size  of  0.72  (roughly,  an  improvement  from  the  50th  to  76th  percentile) 
in  the  ability  to  operate  and  locate  faults  on  a  movie  projector.  Verano  also  compared  an  interactive,  adaptive, 
branching  approach  with  a  strictly  linear  approach  for  presenting  instmctional  material  in  beginning  Spanish. 
He  reported  an  effect  size  of  2.16  (roughly,  an  improvement  from  the  50th  to  98th  percentile)  in  end-of-course 
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knowledge.  These  two  studies,  among  others,  suggest  that  individualization  of  sequence  and  content  matters  - 
perhaps  a  great  deal. 


A.5  TECHNOLOGY-ASSISTED  LEARNING 
A.5.1  Time  Savings 

One  of  the  most  stable  findings  in  the  comparisons  of  technology-based  instruction  and  conventional  instruction 
(which  uses  lecture,  text,  and  experience  with  equipment  (e.g.,  in  the  laboratory))  concerns  instruction  time 
savings.  Table  A-l  presents  these  findings. 


Table  A-1 :  Percent  Time  Savings  for  Technology-Based  Instruction. 


Study 

(Reference) 

Number  of 
Studies 
Reviewed 

Average  Time 
Saved 
(Percent) 

(Military  Training)  [39] 

13 

54 

(Higher  Education)  [13] 

8 

31 

(Higher  Education)  [32] 

17 

34 

(Adult  Education)  [32] 

15 

24 

As  the  table  shows,  Orlansky  and  String  (1977)  [39]  reported  that  reductions  in  time  to  reach  instructional 
objectives  averaged  about  54%  in  13  assessments  of  technology-based  military  training.  Fletcher  (1991)  [13] 
reported  an  average  time  reduction  of  3 1  %  in  8  assessments  of  1M1  applied  in  higher  education.  Kulik  reported 
average  time  reductions  of  34%  in  17  assessments  of  technology  used  in  higher  education  and  24%  in 
15  assessments  of  adult  education  [32].  Each  of  these  reviews  covered  different  sets  of  evaluation  studies  that 
compared  technology-based  instruction  and  conventional  classroom  instruction  involving  lecture,  texts, 
and  perhaps  laboratory  examples.  Overall,  it  seems  reasonable  to  expect  that  technology-based  instruction  will 
reduce  by  about  30%  the  time  students  take  to  reach  a  variety  of  objectives. 

A.5.2  Costs 

An  example  of  the  cost  benefits  of  this  reduction  in  time  to  learn  can  be  seen  in  residential,  specialized  skill 
training.  The  DoD  spends  about  $6.5  billion  a  year  on  this  training,  which  is  the  “schoolhouse”  training 
individuals  receive  after  basic,  or  initial  accession,  training.  It  qualifies  individuals  for  the  many  technical  jobs 
(e.g.,  wheeled  vehicle  mechanics,  radar  operators,  avionics  technicians,  medical  technicians,  and  so  forth) 
needed  to  perform  military  operations.  It  does  not  include  the  costs  of  aircraft  pilot  training,  officer  education, 
or  training  provided  in  military  units. 

Extrapolated  from  an  earlier  analysis  by  Angier,  Fletcher,  and  Horowitz  (1991)  [2],  Figure  A-2  shows  the  annual 
reductions  in  costs  that  would  result  if  instruction  time  were  reduced  by  30%  for  20,  40,  60,  and  80  %  of  military 
personnel  who  complete  residential,  specialized  skill  training  each  year.  For  instance,  if  the  DoD  reduced  by 
30%  the  time  to  train  20%  of  the  personnel  undergoing  specialized  skill  training,  it  would  save  about  $428 
million  per  year.  If  it  were  to  do  so  for  60%  of  these  personnel,  it  would  save  about  $1,284  million  per  year  - 
or  about  20%  of  the  funds  allocated  for  specialized  skill  training.  Specialized  skill  training  is  particularly 
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amenable  to  the  use  of  ADL  technologies.  Use  of  these  technologies  by  60%  of  specialized  skill  trainees  is  not 
an  unreasonable  expectation. 


1800 

1600 

1400 

1200 


c  1000 
o 

i 

W  800 


600 

400 

200 

0 


$1,710 

$1,284 

S8S4 

$428 

20% 


40%  60% 

Percent  of  Students  Covered 


80% 


Figure  A-2:  Cost  Savings  ($2008)  in  Specialized  Skill  Training 
Assuming  a  30%  Reduction  in  Training  Time. 

Saving  30%  of  training  time  may  be  a  conservative  target.  Commercial  enterprises  that  develop  technology- 
based  instruction  for  the  DoD  regularly  base  their  bids  on  the  expectation  that  they  can  reduce  instructional  time 
by  50%.  Noja  (1987)  [38]  has  reported  time  savings  as  high  as  80%  with  the  use  of  technology-based  instruction 
in  training  operators  and  maintenance  technicians  for  the  Italian  Air  Force. 

Two  other  sources  of  cost  savings  with  ADL  technologies  are  not  considered  in  the  Figure  A-2  data. 
First,  the  cost  models  used  to  generate  the  data  in  Figure  A-2  assume  reductions  in  training  time  in  residential 
settings  but  do  not  take  into  account  using  ADL  technologies  to  distribute  some  of  that  training  to  operational 
units  -  thereby  reducing  change  of  station  or  temporary  duty  costs.  Second,  ADL  technologies  can  be  used  to 
simulate  expensive  equipment,  operational  environments,  and  interpersonal  situations  -  thereby  not  just  reducing 
costs,  but  also  increasing  safety,  enhancing  visualization,  and  allowing  time  to  be  sped  up  or  slowed  down  as 
needed  for  the  training. 

Perhaps  more  importantly  for  military  applications,  ADL  technologies  can  prepare  individuals  more  quickly  for 
operational  duty.  In  this  way,  these  technologies  act  as  force  multipliers  by  increasing  readiness  and  operational 
effectiveness  without  increasing  personnel  costs. 
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A.5.2.1  Instructional  Effectiveness 

Research  data  suggest  that  savings  from  using  ADL  technologies  do  not  come  at  the  expense  of  instructional 
effectiveness.  Empirical  findings  report  the  opposite.  Figure  A-3  shows  effect  sizes  from  several  reviews  of 
studies  that  compared  conventional  instruction  with  technology-based  instruction. 


Figure  A-3:  Some  Effect  Sizes  for  Studies  Comparing  Technology-Based 
Instruction  with  More  Conventional  Approaches. 


In  Figure  A-3,  Computer-Based  Instruction  (CBI)  summarizes  the  results  from  233  studies  that  involved  a 
straightforward  application  of  computer  presentations  using  text,  graphics,  and  some  animation,  as  well  as 
some  degree  of  individualized  interaction.  The  effect  size  of  0.39  standard  deviations  suggests,  roughly, 
an  improvement  of  50th-percentile  students  to  the  performance  levels  of  65th-percentile  students. 

Interactive  Multi-media  Instruction  (IMI)  involves  more  elaborate  interactions  adding  audio,  animation, 
and  video  and  generally  taking  advantage  of  the  multi-media  effect  [20],  [34].  These  added  capabilities  evidently 
increase  achievement.  They  show  an  average  effect  size  of  0.50,  which  suggests  an  improvement  of 
50th-percentile  students  to  the  performance  levels  of  69th  pcrccntilc  students. 

Intelligent  Tutoring  Systems  (ITSs)  involve  a  capability  that  has  been  developing  since  the  late  1960s 
(e.g.,  [6]  and  [41]).  In  this  approach,  an  attempt  is  made  to  directly  mimic  the  one-on-one  dialogue  that  occurs  in 
tutorial  inter-actions.  A  key  goal  of  these  systems  is  to  generate  computer  presentations  and  responses  in  real 
time  and  on  demand  as  needed  or  requested  by  learners.  Instructional  designers  do  not  need  to  anticipate  and 
pre-store  them.  This  approach  is  computationally  more  sophisticated  and  more  expensive  to  produce  than 
standard  computer-based  instruction.  However,  its  cost  can  be  justified  by  the  increase  in  average  effect  size 
to  0.84  standard  deviations,  which  suggests,  roughly,  an  improvement  of  50th-percentile  students  to  the 
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performance  levels  of  80th-percentile  students.  As  will  be  discussed  later,  ROl  is  much  more  sensitive  to 
scaling  and  delivery  costs  of  instruction  than  to  the  initial  costs  to  design  and  develop  it. 

A  selected  group  of  ITSs  (“Recent  Intelligent  Tutors”)  was  considered  just  to  see  how  far  these  systems  are 
progressing.  The  average  effect  size  of  1.05  standard  deviations  for  these  applications  is  promising.  It  represents, 
roughly,  an  improvement  of  50th-percentile  students  to  the  performance  levels  of  85th-percentile  students. 

The  extensive  tailoring  of  instruction  that  generative,  ITSs  provide  to  meet  the  needs  of  individual  students  can 
only  be  expected  to  increase.  Such  systems  may  raise  the  bar  -  well  past  Bloom’s  2-Sigma  challenge  -  for  the 
ultimate  effectiveness  of  ADL-based  instruction. 

A.5.2.2  Student  Attitudes 

Student  attitudes  toward  instruction  can  affect  its  effectiveness  and  efficiency.  Many  evaluations  of  technology- 
based  instruction  simply  ask  students  if  they  prefer  it  to  more  conventional  classroom  approaches.  Greiner 
(1991)  [29]  reviewed  these  evaluations  and  found  overall  that  70  -  80  %  of  students  who  were  polled  preferred 
technology-based  approaches  to  other  approaches.  When  students  reported  that  they  did  not  prefer  the 
technology-based  approaches,  the  reasons  were  usually  because  of  implementation  or  technical  problems  with 
the  technology  and  not  the  instructional  approach  itself. 

McKinnon,  Nolan,  and  Sinclair  (2000)  [35]  completed  a  thorough  3-year  study  of  student  attitudes  toward  the 
use  of  technology-based  learning  and  productivity  tools  such  as  spreadsheets,  databases,  graphics,  desk-top 
publishing,  and  statistical  processing.  The  attitudes  of  the  students  toward  technology  use  slackened  as  the 
novelty  of  using  the  technology  wore  off.  However,  their  attitudes  remained  positive  and  significantly  more 
positive  than  those  of  students  who  did  not  have  access  to  the  technology  throughout  the  3  years  of  the  study. 

A.5.2.3  Return  On  Investment  (ROI) 

Knowing  that  we  can  use  ADL  technologies  to  reduce  learning  time,  particularly  time  to  learn  journeyman  skills 
such  as  remembering,  understanding,  and  applying  facts,  simple  concepts,  and  straight-forward  procedures, 
what  might  an  investment  in  these  technologies  return? 

One  way  to  answer  this  question  is  by  applying  the  findings  presented  earlier  in  Figure  A-2,  which  only 
considered  savings.  Using  the  analysis  underlying  that  figure,  we  can  wrap  in  both  the  savings  and  the  costs  to 
achieve  them  using  an  ROl  model.  This  model  simply  reduces  to  the  ratio  of  the  net  return  (savings  in  this  case) 
to  the  costs  as  shown  in  the  following: 

Savings  -  Costs 

Costs 

We  can  begin  by  assuming  (conservatively)  a  30%  reduction  in  training  time  achieved  through  the  use  of  ADL 
technologies  by  (conservatively)  40%  of  residential  specialized  training  students.  From  Figure  A-2  and  the 
analysis  on  which  it  is  based  [2],  these  assumptions  suggest  annual  savings  of  $854  million.  Given  this  result, 
the  next  step  is  to  determine  the  costs  to  design,  develop,  and  deliver  ADL  instruction  to  40%  of  residential 
specialized  skill  training  students. 

Using  ADL  technology,  how  much  would  it  cost  to  render  the  training  needed  by  40%  of  specialized 
skill  students?  According  to  the  last  published  Military  Manpower  Training  Report  (2002)  [36],  the  average 
Specialized  Training  course  length  across  all  four  Services  was  about  57  training  days.  If  that  number  were 
reduced  by  30%,  the  training  course  length  would  amount  to  about  40  days.  Military  personnel  in  training  billets 
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may  be  required  to  perform  additional  duties,  but,  assuming  8  hours  a  day  in  course  training  amounts  to  about 
320  hours  of  training.  About  357,700  officers  and  enlisted  personnel  completed  Specialized  Training  in  FY 
2002.  Forty  percent  of  that  number  amounts  to  143,080  learners.  In  effect  then,  320  hours  of  ADL  training 
would  have  to  be  produced  and  then  delivered  to  143,080  learners. 

Estimates  to  produce  an  hour  of  computer-assisted  training  vary  widely  and  depend  on  the  content,  instructional 
strategy,  and  pay  and  allowances  for  subject-matter  experts,  “authors,”  and  computer  programmer/analysts. 
One  source  of  estimates  comes  from  the  Joint  Knowledge  Development  and  Distribution  Capability  (JKDDC), 
which  is  producing  web-based  individual  training  programs  for  joint  assignments  and  operations.  As  of 
May  2008,  JKDDC  had  produced  over  200  courses,  with  more  than  65,000  course  completions  [5].  In  the  first 
quarter  of  FY  2008,  the  costs  for  JKDDC  to  develop  an  hour  of  instruction  were  about  $14,000,  and  the  costs  to 
deliver  it  were  about  $4. 

We  might  then  assume  that  it  would  cost  about  $4.48  million  to  produce  320  hours  of  ADL  specialized  skill 
training  and  an  additional  $183.14  million  to  deliver  it  to  143,080  students.  The  investment  for  the  first  year  of 
this  training  would  cost  about  $187.62  million. 

Developing  and  delivering  technology-based  instruction  to  40%  of  Specialized  Skill  students  (about  143,080 
students)  would  require  30%  fewer  hours  to  complete  their  training  -  a  savings  of  about  136  hours  for  each 
student.  Assuming  a  composite  of  $42  per  hour  in  pay  and  allowances  [37],  this  use  of  technology-based 
instruction  would  amount  to  (42  x  143,080  x  136)  =  $817.27  million.  Plugging  these  data  and  assumptions  into 
an  ROI  calculation  yields: 

($817.27  M  -  $187.62  M)  /  $187.62  M  =  $3.36 

Under  these  assumptions,  an  investment  in  ADL  technology  will  return  about  $3.36  for  every  dollar  invested. 

After  the  first  year,  the  costs  to  develop  the  instruction  would  be  reduced  to  whatever  is  required  to  maintain  and 
update  the  course,  but  the  ROI  is  not  particularly  sensitive  to  development  costs.  It  seems  far  more  sensitive  to 
delivery  costs.  They  are  included  in  this  analysis  for  the  delivery  of  the  ADL  instruction  but  not  for  the 
classroom  instruction  it  replaces.  If  the  savings  in  training  delivery  costs  were  fully  considered  in  this 
calculation,  the  ROI  woidd  increase  substantially.  Also,  as  suggested  earlier,  30%  time  savings  is  likely  to  be  an 
underestimate  of  the  student  time  in  training  that  can  be  saved.  Further,  even  though  this  analysis  assumes  web- 
based  capability  for  delivering  the  instruction,  it  does  not  take  into  account  reductions  in  travel  and  temporary 
duty  costs,  which,  if  included,  would  further  increase  ROI.  Finally,  the  administrative  efficiencies,  improved 
tracking  and  assurance  of  student  progress,  and  other  benefits  provided  by  ADL  technology  -  benefits  that  were 
not  considered  in  the  previous  analysis  -  have  their  place  and  would  continue  to  argue  for  its  use. 

Less  time  in  school  means  more  time  on  the  job.  Savings  in  time  needed  to  reach  training  objectives  not  only 
reduce  training  costs,  but  also  increase  the  supply  of  people  for  operational  forces  without  increasing  the  number 
of  people  in  uniform.  Ways  to  account  for  accompanying  increases  in  readiness  and  effectiveness  due  to  force 
multiplication  remain  to  be  determined,  but  they,  rather  than  savings  in  training  costs,  may  be  the  most 
significant  impact  of  reducing  the  time  required  by  operational  forces  to  reach  performance  levels. 

In  short,  a  significant  return  seems  likely  to  result  from  an  investment  to  convert  some  proportion  of  training 
to  ADL  technology.  This  value  arises  primarily  from  the  reduction  of  student  time  spent  in  the  training 
infrastructure.  Even  with  ADL,  seasoned  military  personnel  must  provide  additional  training,  mentoring, 
and  monitoring  for  people  early  in  their  careers.  Certainly,  efforts  must  be  made  to  preserve  the  camaraderie  and 
esprit  de  corps  gained  by  students  undergoing  the  rigors  of  training  together.  The  argument  for  ADL  is  not  to 
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suggest  the  massive  replacement  of  people  by  the  technology.  It  is,  instead,  that  ADL  reduces  costs  to  train  and 
increases  force  effectiveness  by  releasing  people  sooner  from  the  training  infrastructure  and  ensuring  their 
competencies.  A  cost-  effective,  optimal  balance  between  the  use  of  ADL  and  more  people-intensive  approaches 
to  training  and  education  remains  to  be  determined. 

A.5.3  Performance  Aiding 

Most  of  the  discussion  to  this  point  has  focused  on  education  and  training  applications  using  ADL  technology. 
Something  remains  to  be  said  about  its  use  in  providing  on-  demand  performance  aiding.  The  term  Interactive 
Electronic  Technical  Manual  (1ETM)  is  a  generic  label  for  such  a  device  [22].  Fletcher  and  Johnston  (2002)  [18] 
presented  data  on  effectiveness  and  costs  of  several  portable,  electronic  maintenance  performance  systems, 
one  of  which  was  the  Integrated  Maintenance  Information  System  (IMIS). 

IMIS  was  a  wearable,  computer-based  performance  aid  for  avionics  maintenance.  Thomas  (1995)  [45]  compared 
the  performance  of  12  Avionics  Specialists  and  12  Airplane  General  (APG)  Technicians  in  troubleshooting  three 
F-16  avionics  sub-systems.  Within  each  of  the  two  groups  of  subjects,  six  of  the  fault  isolation  problems  were 
performed  using  paper-based  Technical  Orders  (TOs)  (Air  Force  technical  manuals),  and  six  were  performed 
using  IMIS. 

Training  for  APG  Technicians  includes  all  aspects  of  aircraft  maintenance,  only  a  small  portion  of  which 
concerns  avionics.  In  contrast,  Avionics  Specialists  must  meet  higher  selection  standards  and  receive  16  weeks 
of  specialized  training  that  focuses  on  avionics  maintenance.  Costs  to  train  APG  Technicians  are  about  half  the 
costs  for  Avionics  Specialists.  Table  A-2  shows  the  results  of  the  study. 

Table  A-2:  Maintenance  Performance  of  12  Air  Force  Avionics 
Specialists  and  12  APG  Technicians  Using  TOs  and  IMIS. 


Correct  Time  to  Average  Time  to 

Technicians/Performers  Solutions  Solution  Number  of  Order  Parts 

(Percent)  (Minutes)  Parts  Used  (Minutes) 


TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

TOs 

IMIS 

Avionics  Specialists 

81.9 

100.0 

149.3 

23.6 

8.7 

6.4 

19.4 

1.2 

APG  Technicians 

69.4 

98.6 

175.8 

124.0 

8.3 

5.3 

25.3 

1.5 

As  shown  in  the  table,  findings  of  the  study  were  as  follows: 

•  Avionics  Specialists  using  TOs  compared  with  those  using  IMIS  -  The  Avionics  Specialists  using 
IMIS  found  more  correct  solutions  in  less  time,  used  fewer  parts  to  do  so,  and  took  less  time  to  order  the 
parts.  All  these  results  were  statistically  significant.  The  results  concerning  time  to  order  parts  are  to  be 
expected  because  IMIS  automated  much  of  this  process. 

•  APG  Technicians  using  TOs  compared  with  those  using  IMIS  -  The  APG  Technicians  using  IMIS 
found  more  correct  solutions  in  less  time,  used  fewer  parts  to  do  so,  and  took  less  time  to  order  the  parts. 
All  these  results  were  statistically  significant. 

•  APG  Technicians  using  IMIS  compared  with  Avionics  Specialists  using  TOs  -  The  APG 

Technicians  using  IMIS  found  more  correct  solutions  in  less  time,  used  fewer  parts  to  do  so,  and  took 
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less  time  to  order  the  parts  than  did  Avionics  Specialists  using  paper-based  TOs.  All  these  results  were 
statistically  significant. 

•  APG  Technicians  using  IMIS  compared  with  Avionics  Specialists  using  IMIS  -The  APG 
Technicians  performed  just  about  as  well  as  the  Avionics  Specialists  and  even  slightly  better  in  the 
number  of  parts  used.  None  of  these  comparisons  were  statistically  significant. 

The  economic  promise  suggested  by  these  results  could  well  vanish  if  the  costs  to  provide  the  performance  aid 
(i.e.,  the  IMIS)  exceed  the  costs  they  otherwise  save.  Teitelbaum  and  Orlansky  (1996)  [46]  estimated  IMIS 
reductions  in  depot-level  maintenance,  organizational-level  maintenance,  and  maintenance  and  transportation  of 
inventories  of  spare  parts.  They  found  annual  savings  from  the  use  of  IMIS  at  about  $38  million  for  the  full  Air 
Force  fleet  of  F-16s.  They  assumed  an  8-year  useful  life  for  IMIS  and  estimated  about  $18  million  per  year  to 
maintain  and  update  it  and  amortize  its  development  costs. 

The  result  is  a  benefit  of  about  $20  million  per  year  in  net  savings  or  an  ROl  of  about  1.11,  which  excludes  the 
significant  impact  of  IMIS  on  sortie  rate,  readiness,  and  operational  effectiveness. 


A.6  THE  ADL  VISION 

The  long-term  ADL  vision  is  an  extrapolation  from  such  developments  as  portable,  increasingly  accessible 
computing  (including  hand-held,  worn,  or  even  implanted  computers),  the  global  information  infrastructure 
(currently  manifest  in  the  World  Wide  Web  with  its  multifarious  search  engines),  modular  object-oriented 
architectures,  Web  2.0  technologies,  and  natural  language  processing.  The  march  toward  devices  that  might  be 
described  as  personal  learning  associates  seems  inevitable. 

As  currently  envisioned,  these  devices  will  act  as  personal  accessories.  They  will  respond  on  demand  to  each 
individual’s  needs  for  education,  training,  and  performance  aiding  by  assembling  relevant  objects  from  the 
global  infrastructure  and  engaging  the  user  in  guided  conversations,  such  as  those  described  by  Hu,  Graesser, 
and  Fowler  [30],  to  enhance  user  knowledge  and  skills  and/or  problem-solving  capabilities.  Learning  in  these 
cases  is  not  a  matter  of  just  working  through  pre -specified  lessons  but  is  a  return  to  the  100,000  year  old  tutorial 
practice  of  an  individual  and  a  sage  working  together  to  enhance  knowledge  and  skill.  In  this  case,  the  human 
sage  is  supplanted  by  a  computational  device  that  has  access  to  something  approaching  the  whole  of  human 
knowledge  carried  throughout  the  global  information  infrastructure. 

As  described  Dodds  and  Fletcher  (2004)  [12]  and  by  Wisher  [50],  objects  drawn  from  the  global  infrastructure 
must  be  portable,  durable,  reusable,  and  accessible.  Gallagher  (in  press)  [24]  discusses  the  history  and 
development  of  the  Sharable  Content  Object  Reference  Model  (SCORM),  which  can  ensure  that  objects  have  the 
first  three  of  these  qualities. 

In  addition,  the  objects  must  be  accessible.  As  discussed  in  more  detail  by  Lannom  [33],  the  development  of  the 
Content  Object  Repository  Registration/Resolution  Architecture  (CORDRA)  and  its  use  by  the  ADL  registry 
provide  global  visibility  for  objects  while  allowing  their  developers  to  retain  control  over  access  to  them. 
They  are  then  available  for  reuse  or  repurposing. 

These  developments  have  provided  ways  for  objects  to  be  identified  and  collected  for  local  use  from  the  global 
information  infrastructure.  The  continued  operation  of  Moore’s  Law  and  the  market-driven  effort  to  imbue 
computer  technology  with  natural  language  understanding  should  ensure  development  of  affordable,  mobile, 
conversation-capable  computing.  The  major  issue  that  remains  for  ADL  is  development  of  the  envisioned 
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individualized  tutorial  capabilities.  Progress  and  promise  can  be  seen  in  this  area  (e.g.,  [28]  and  [30]),  and  it 
remains  key  for  realizing  the  full  ADL  vision. 


A.7  CONCLUSIONS 

In  short,  the  aforementioned  research  suggests  that  effective  use  of  ADL  technology: 

•  Increases  instructional  effectiveness; 

•  Reduces  time  needed  to  leam; 

•  Ensures  that  all  students  leam; 

•  Is  preferred  by  students;  and 

•  Is  effective  and  efficient  for  distributing  instruction  anytime,  anywhere. 

Most  of  research  and  data  to  support  these  conclusions  have  been  available  for  some  time.  The  usual  lag  between 
research  findings  and  their  application  in  practice  is  observable  here  as  elsewhere.  As  argued  first  by  Fletcher 
(1992,  1997)  [14],  [15]  and  later  by  Corbett  (2001)  [8],  ADL  technology  may  make  Scrivin’s  educational 
imperative  and  Bloom’s  tutorial  instruction  affordable. 

The  “Rule  of  Thirds”  that  emerges  from  empirical  evaluations  of  technology-based  instruction  was  mentioned 
earlier  in  this  report.  The  Rule  of  Thirds  is  strictly  a  statistical  statement.  It  summarizes  a  large  body  of  empirical 
findings,  but  it  does  not  directly  address  cause  and  effect. 

An  often  quoted  statement  that  “The  best  current  evidence  is  that  media  are  mere  vehicles  that  deliver  instruction 
but  do  not  influence  student  achievement  any  more  than  the  truck  that  delivers  our  groceries  causes  changes  in 
our  nutrition”  ([7],  p.  445)  does  address  cause  and  effect.  This  point  of  view  seems  both  fair  and  unequivocal. 
The  presence  of  any  technology  is  no  guarantee  that  effective  instructional  content,  effective  ways  to  present  it, 
or  even  that  the  unique  strengths  of  the  technology  itself  will  be  present  or  used.  On  the  other  hand,  the  absence 
of  technology,  including  ADL  technology,  is  a  reasonable  guarantee  that  its  unique  functionalities  will  be 
unavailable. 

Another  statement  to  consider  is  that  “If  you  don’t  have  a  gadget  called  a  ‘teaching  machine,’  don’t  get  one. 
Don’t  buy  one;  don’t  borrow  one;  don’t  steal  one.  If  you  have  such  a  gadget,  get  rid  of  it.  Don’t  give  it  away, 
for  someone  else  might  use  it.  .  .  .  If  you  begin  with  a  device  of  any  kind,  you  will  try  to  develop  the  teaching 
program  to  fit  that  device”  ([26]  p.  478). 

Gilbert  seems  both  right  and  wrong.  He  is  certainly  correct  in  suggesting  that  instructional  designers  and 
developers  who  adapt  a  “teaching  machine”  will  try  to  fit  the  teaching  program  to  it.  However,  the  new 
functionalities  such  a  device  makes  available  motivate  its  adoption  and  instructional  adaptations  to  it. 

It  is  less  certain  that  such  adaptations  are  to  be  avoided.  They  might  well  be  enthusiastically  sought,  just  as 
printed  textbooks  were  sought  long  ago.  If  properly  applied,  technology  should  improve  -  if  not  revolutionize  - 
the  effectiveness  and  efficiency  of  teaching  programs.  It  is  up  to  researchers,  developers,  and  instructors  - 
not  the  technology  itself  -  to  see  that  it  does. 

Finally,  there  is  the  Columbus  effect.  In  keeping  with  technologies  that  made  carriages  go  without  horses  and 
telegraphs  transmit  without  wires,  the  Columbus  effect  will  doubtless  apply  to  our  efforts  to  provide  tutorial 
instruction  without  humans.  We  envision  the  development  of  personal  learning  associates  and  are  building 
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toward  them.  However,  just  as  Columbus  headed  for  the  East  Indies  and  ended  up  with  something  entirely 
unexpected,  we  may  end  up  with  something  as  unforeseen  and  different  from  horseless  carriages  and  wireless 
telegraph  as  automobiles  and  radio.  Nonetheless,  making  education,  training,  and  problem-solving  aids  as 
affordable  and  universally  accessible  as  possible  seems  as  good  a  start  as  any. 
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